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ABSTRACT 



This report contains the results of a review process and 
specifically addresses the indicators of progress and achievement which 
include student impact, teacher impact, policy changes, resource changes, 
management change, data utilization, learning infrastructure change, student 
performance, and partnerships. The review was designed to gain a full 
understanding of the appropriately- documented, effectively-measured, 
significant, and reliable indicators by which progress is assessed according 
to the Statewide Systemic Initiative (SSI) Program objectives. (DDR) 
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xi. 1998 Program Effectiveness Reviews (PEiff Report 
SystemicStifaiatives (SSI) program 

■ SSI Program 

Program Effectiveness Reviews. With congressional passage of the Government 
Performance and Results Act (GPRA) of 1993, starting in FY1999, the National Science 
Foundation (NSF) budget requests must be outcomes-based. For the Directorate for 
Education and Human Resources (EHR) portfolio, it must be demonstrated how the 
Directorate’s activities and programs fulfill the relevant goal of the NSF Strategic Plan, 
“excellence in science, mathematics, engineering and technology education at all levels”, 
and yield value to the American public. Moreover, the recently initiated examination of 
the science, mathematics, engineering, and technology portfolio by the Education and 
Human Resources Committee of the National Science Board and the increased interest by 
the Administration and Congress in demonstrated program effectiveness, have also 
contributed to this time of measurement, interpretation, documentation of the progress of 
the projects funded, and the call for more explicit and timely responses to questions that 
bear on the returns on investments made under the rubric of EHR’ s programming. 

The Program Effectiveness Reviews (PER) specifically addressed the following indicators 
of progress/achievement: 

(i) student impact, 

(ii) teacher impact, 

(iii) policy changes, 

(iv) resource changes, 

(v) management change, 

(vi) data utilization, 

(vii) learning infrastructure change, 

(viii) student performance, and 
(ix) partnerships. 

The Statewide Systemic Initiative (SSI) Program is one of a select number of programs 
subjected to the effectiveness review, which focuses on project (in the aggregate program) 
outputs, outcomes, and impacts. The review was designed to gain a full understanding of 
the appropriately documented, effectively measured, significant, and reliable indicators by 
which progress is assessed as per the SSI Program objectives, design and implementation 
strategies, and the financial resources expended. 

SSI Program Objectives and Design: The President and governors have articulated an 
ambitious set of national goals, including the goal of making the United States first in the 
world in mathematics and science by the year 2000. Achieving these goals require wide- 
ranging changes in both policy and practice. 

The NSF believes that state leadership is critical to reform efforts in schools. States not 
only have the constitutional authority for education, but they are the only level of the 




XJ 



education system that can influence all of the K-12, as well as many aspects of 
postsecondary education, such as teacher preparation, certification, and licensure. They 
are in a position to coordinate resources from a variety of public and private sources and 
change regulations and policies. In addition, they are able to allocate resources across the 
state fairly and equitable. 

The mission of the EHR’s Division of Educational System Reform (ESR) is to stimulate 
the reform of science and mathematics education in the United States so that every student 
experiences challenging instruction that allows his or her abilities to be fully developed and 
that prepares her or him well for life and work in an increasingly technological world in 
the years to come. This responsibility is national in its dimensions, even though NSF’s 
role is essentially catalytic. Most of the resources that will be required to attain systemic 
reform objectives, if the mission is to be realized, will come from sources other than NSF. 
Nevertheless, the Foundation desires to effectively leverage those resources to enable a 
unitary program of high-quality, equitable education for each student in the Nation. 

Consequently, the NSF solicited proposals by states intended to generate comprehensive 
systemic reform designed to achieve significant improvements in science, mathematics, 
and engineering education. Also, these reform efforts are aimed at broadening the impact, 
accelerating the pace, and increasing the effectiveness of improvements in science, 
mathematics, and engineering education in both K-12 and post-secondary levels. The 
anticipated effects of these changes are: 

(i) to increase the knowledge of science and mathematics acquired by all students as all 
educational levels; 

(ii) to afford every student the maximum opportunity to acquire the habits of mind and 
critical thinking skills that characterize effective use of mathematics and science; and 

(iii) to enable students to understand the role and influence of technology as one of a 
number of ways by which science and mathematics are related to the physical world 
and the human condition. 

The NSF established the SSI Program in 1990 as a program based on three cohorts of 
five-year awards. It was designed to encourage more coherent and consistent policies and 
programs in participating states by asking states to identify the elements that, taken 
together, would make a difference in what students know and are able to do. States were 
expected to integrate into effective plans such components of systemic change as: 

(i) organizational structure and decision making; 

(ii) provision and allocation of resources; 

(iii) recruitment and preparation of teachers and college faculty; 

(iv) retention and continuing professional development of teachers and other professional 
personnel; 

(v) curriculum content and learning goals; 

(vi) delivery of instruction, including the use of educational technology; 

(vii) assessment of student achievement; 

(viii) facilities and equipment; 

(ix) articulation within the system, and; 
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(x) accountability systems. 

To be considered systemic, states had to identify how they would develop and manage. 

(i) state commitment in terms of resources and focus; 

(ii) the state's vision of mathematics and science education; 

(iii) the creation of partnerships to enable the effort to succeed; and 

(iv) a plan for implementing and evaluating results for both management purposes and 
those of accountability. 

Chronology of Awards and Total Expenditures: As a result, a total of 25 states and the 
Commonwealth of Puerto Rico received SSI awards for terms of up to 5 years of funding 
and of up to approximately $2.0 million per year. The Foundation has provided a total of 
approximately $265 million to date to stimulate and catalyze selected, high-quality, 
systemic reform efforts designed to move quickly and purposefully to action as follows. 



F 


Sites • ■ 


Expenditures 


FY 91 
Cohort I 


Connecticut, Delaware, Florida^pui^iatia, Montana* , 
Nebraska, North Carolina, Ohio, Rhode Island, and South 
Dakota Nv- Y 


$14,366,266 


FY9 
Cohort II 


California:, 

Michigan, New Mexico, Puer%Rico, Texas,- Vermont;! and 
Virginia 




FY 93 
Cohort III 


Arkansas, Colorado- New Jersey, New Y ork, and South 
Carolina ■ ■■ v.V- • 


$53,693,043 


FY 94 




$54,351,619 


FY 95 




$44,380,524 


FY 96 




$38,570,994 


FY 97 




$18,092,886 



Management and Oversight: The SSIs and the NSF signed cooperative agreements to 
ensure the accomplishment of the common agreed upon goals. The initiatives are required 
to submit annual progress reports, including data reporting. The SSIs have been 
supported by technical assistance, evaluation, and monitoring contracts. In addition, every 
site went through the Mid-Point Review process and each site has received at least one 
site visit each year. 

Cooperative Agreements : Cooperative agreements are used by NSF for awards when the 
accomplishment of the project objectives requires substantial NSF technical or 
management involvement during performance of the activities. Substantial NSF 
involvement may be necessary when an activity is technically or managerially complex, or 
requires extensive or close coordination with other Federally supported work. 

Systemic Reform Critical Developments : In 1995, the NSF disseminated the Instrument 
for Annual Report of Progress in Systemic Reform , a guiding document that includes a 
description of the critical developments that drive systemic reform (drivers). It is used for 
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various purposes in the SSI Program, including planning, evaluation, and reporting of the 
various initiatives. The six drivers are: 

(i) Implementation of comprehensive, standards-based curricula; 

(ii) Development of a coherent, consistent set of policies that support high-quality 
mathematics and science education; 

(iii) Convergence of the usage of all resources into a focused and unitary program; 

(iv) Broad-based support from parents, policymakers, institutions of higher education, 
business and industry; 

(v) Accumulation of a broad and deep array of evidence that the program is enhancing 
student achievement; and 

(vi) Improvement in the achievement of all students, including those historically 
underserved. 

Mid-Point Reviews : Mid-Point Reviews (MPRs) are standard procedure in EHR/NSF for 
large-scale awards, particularly helpful in providing information on programs funded oyer 
long periods of time. The goal of these reviews is to determine the impact of the SSI 
awards to date, and to ascertain their potential to stimulate improvements in the future. 
MPRs consisted of panels of outside experts who examined the accomplishments of the 
SSI states and, consequently, made recommendations about future directions and funding 
continuation for those states. These reviews are usually followed by site visits. The 
results of the site visits and the panels’ recommendations are shared as recommendations 
with the initiatives’ leadership. 

Technical Assistance : ESR provides technical assistance to all SSIs. This service begins 
with the Program Officer's analysis of the initiatives’ annual reports and proposed strategic 
plans. Technical assistance needs are identified, discussed with the sites, and 
communicated to the technical assistance contractor, who provides services in three major 
domains. 

(i) site-based technical assistance, 

(ii) communications network, and 

(iii) leadership development. 

Annual Reports : SSIs submit annual progress reports (usually by May 15) in which they 
describe their major accomplishments in terms of the drivers, their impact, outcomes and 
strategic plans for the upcoming year. Also, sites submit data reports twice a year. 

Evaluation : The National Evaluation of the SSI Program, conducted by the external 
contractor, SRI International, assessed the SSI as a federal strategy for improving 
mathematics and science education by documenting educational changes at each stage of 
the systemic reform process: project activities, in-state-level collaboration, state policy and 
practice, local and higher education delivery and policy systems, and student experience 
and outcomes. SRI has conducted field work in various SSI states that has resulted in 
reports, such as SSI Evaluation Reports, Public Case Studies, Compilation of State Data 
from the Annual Reports to NSF; as well as other resource documents, such as The SSI 
Program 's Impact on Students: What We Now Know and What We Expect to Learn, The 
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Impact on Students of the SSI Program: A Pilot Study of the Impacts of the Louisiana 
and Montana SSIs. 

Monitoring : Abt Associates Inc. provided monitoring services to the SSI states from 
1992 to 1996. Monitors visited the sites twice a year and focused their work on issues 
related to the implementation of the science and mathematics systemic reform plans and 
the analysis of the factors that contribute to the attainment of the initiatives’ objectives. In 
addition, monitoring services focused on specific issues such as professional development, 
equity, standards, quality control, scaling-up, and sustainability. Reports on progress of 
implementation were developed and shared with Program Officers and the sites’ 
leadership. 

Site Visits : Program Officers usually conduct visits to the sites to determine evidence of 
progress made by the initiatives and to address particular issues. Each site is subjected to 
at least one visit every year. 

Phase-out process: As a result of the SSI management and oversight system, the NSF 
determined that some initiatives had not achieved an acceptable level of development in 
their science and mathematics education reform efforts. Therefore, SSI awards were 
phased-out in Rhode Island in FY 1994; Florida in FY 1995; and North Carolina and 
Virginia in FY 1996. 

Phase II (Continuation) process: EHR initiated a competitive process in FY 1996 
through which further support is provided to a few exemplary members of each SSI 
cohort to extend their successful results throughout the state with the expectation of 
realizing a fully mature reform initiative. EHR will also continue to work with any of the 
members of each cohort desiring such assistance by providing advice regarding possible 
continuation of individual components of their systemic initiative within the framework of 
EHR's existing programs. Two Cohort I sites, Louisiana and Connecticut, were 
awarded grants in FY 1996 to continue their reform efforts for up to 5 additional years of 
funding of up to $1.4 million per year and totaling under $6 million. In FY 1997, NSF 
opened this competition to the 10 Cohort II remaining members, including California, 
Georgia, Kentucky, Maine, Massachusetts, Michigan, New Mexico, Puerto Rico, 
Texas, and Vermont. Consequently, Massachusetts, Puerto Rico, Texas and Vermont 
were recommended to continue their systemic reform efforts for a similar number of years. 
In FY 1998, Cohort III SSI states, including Arkansas, Colorado, New Jersey, New 
York, and South Carolina will compete for this purpose. The NSF expects to select 
two-three states. 

As the SSIs matured, the focus of the program shifted from the development of cases or 
even exemplars of how K-12 science and mathematics education should be provided by a 
state. The focus became the mission defined earlier: the comprehensive reform of science 
and mathematics education in the United States. The two principal implications of this 
mission are: 

(i) programs must be constructed that can reach most of the students in the nation, and 
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(ii) the results of the programs must be demonstrable— that is, impact on student 
attainment is critical. 

In this regard, NSF does not provide funding unless a state can demonstrate both positive 
impact on student mathematics and science achievement attributable to the SSI and other 
associated and leveraged reform activities; evidence of progressive impact of the SSI, 
upon the state; and a compelling plan for completing scale-up to the entire state. 

: ;} : f I|. 1998 SSI PER Major Findings and Projected Actions 

The 1998 SSI PER was held on December 17, 1997 at the NSF’s headquarters in 
Arlington, Virginia. The process included the states of Connecticut and Louisiana 
(Cohort I, Phase II states); Massachusetts, Puerto Rico, Texas and Vermont, (Cohort 
n, Phase II states); and Arkansas, Colorado, New Jersey, New York, and South 
Carolina (Cohort III states). For the purpose of this report, the Colorado SSI has been 
excluded, because it was considered that this Initiative did not address the GPRA 
indicators in the 1998 PER report; thus, not providing the information and evidence 
requested by this process. 

The following are the major findings and projected actions of the 1998 PER process 
derived from the SSI written reports and oral presentations. They are organized by 
indicators: 

1. Student impact: The average percentage of students impacted by the SSI Program, 
particularly by full implementation of standards-based science and mathematics curricula 
or activities directly related to curriculum implementation in the ten SSI states 
participating in the 1998 PER process was approximately 51% of the total K-12 student 
population in those states. The Puerto Rico SSI reached 200 schools in its first phase or 
12% of the baseline of 1600. By working directly with an additional 200 in 1997-98, the 
Initiative will be reaching 25% of the public schools. During the 1997-98 school year, 
Puerto Rico projects to impact 65,000 students. It and has impacted directly or indirectly 
approximately 600,000 or 92% of all students over the last six years. Likewise, in 1997, a 
total of 657,841 students, or 70% of the total number of public school students in 
Massachusetts, were enrolled in SSI districts. Approximately 80% of these students, or 
56% of all the students in the State, are experiencing a standards-based, inquiry-centered 
mathematics and science program. Similarly, the New Jersey SSI reported a total of 
740,220 students (64% statewide) from districts with a demonstrated high level of 
commitment to standards-based reform, and 192,468 students (17% statewide) from 
districts with mid-level involvement for a total statewide 81% of students impacted in 
1997 In addition, the Arkansas SSI informed that 87% of the 456,000 K-12 students in 
the State have been impacted through participating schools. Of the 1110 schools in the 
State, 80% or 982 schools are SSI schools and of the 3 1 1 school districts, 305 participate 
in one or more SSI project. The remaining SSI states reported the following percentages 
of students impacted: Connecticut-46%; Vermont-43%; Louisiana-3 8%; South 
Carolina-3 1%; Texas-20%; and New York-1. 5%. 
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2. Teacher impact : Overall, the SSI Program impacted approximately 40% of the K-12 
science and mathematics teachers in the reporting states through professional development 
programs of varied duration and intensity, and technical assistance provided to schools 
and school districts. New Jersey, for example, impacted 29,000 mathematics and science 
teachers (72% statewide).from districts with a demonstrated high level of commitment to 
standards-based reform, and over 6,500 teachers (16% statewide) from districts with mid- 
level involvement for an approximate 88% of teachers impacted statewide. Likewise, the 
Massachusetts SSI informed that its teachers have received over 1.3 million hours of 
professional development between 1992 and 1997. Of the 45,325 science and 
mathematics teachers in the State, 31,945 or 70% are in SSI districts. Of these, 27% have 
received professional development of 60-300 hours. An additional 37% of the teachers 
have attended 10-60 hours of mostly district-based professional development to strengthen 
their skills and knowledge of the curriculum frameworks. Moreover, Vermont has 
impacted 52% of its mathematics and science teachers through technical assistance 
provided by teacher associates and college courses as well; while Arkansas has assisted 
approximately 50% of its teachers by means of the “Crusades” (graduate courses offered 
to elementary and secondary teachers). Similarly, South Carolina impacted 30% of its 
mathematics and science teachers through the professional development services provided 
by the 13 regional hubs and the teacher specialists who offer direct assistance to schools 
and districts. Also, Puerto Rico, Connecticut, Louisiana, and Texas impacted 26%, 
22%, 17%, and 7.3% of its teachers, respectively, in 1997, via various strategies, including 
regional demonstration centers, direct service to school districts, college courses and 
specialized workshops. The New York SSI, besides, reported that according to the State 
Education Department’s 1997 Professional Development Surveys , at least 90.6% of the 
teachers who teach in its 97 Research and Demonstration schools have participated in 
awareness-level activities. 

3. Policy changes: SSI states have established a significant range of new strong and 
powerful policies and measures in support of science and mathematics education that have 
not only strengthened the program, but have assisted in directing its evolution as well. 
Arkansas , for example, was successful in passing the Arkansas Public Education Act of 
1997, which establishes one core curriculum, an accountability system, and provisions for 
testing and assessment of standards-based curricula. The legislation is based on the 
premise that all students can and must meet high-quality educational standards if they are 
to be adequately prepared to enter the workforce. In Vermont, Act 60, The Equal 
Education Opportunity Act (1997) drives reform in the State by requiring many aspects of 
improvement: it strengthens and supports the Initiative’s equity efforts, due to the financial 
and programmatic implications; schools must develop annual action plans and reports; 
there is a call for implementation of a statewide K-16 professional development plan. In 
addition, the State has recently approved a new mandated assessment system that includes 
mathematics and science testing. In South Carolina, high school graduation requirements 
were raised from three to four in mathematics and from two to three in science; 
Massachusetts added new teacher certification tests; and the State Board of Education in 
New Jersey unanimously adopted core' curriculum standards with provisions for equity 




C 






7 



planning as a result of the SSI. In addition, in New York, the Regents Competency Tests 
are currently being phased-out and the SSI spearheads the development of assessments 
aligned with standards. In Puerto Rico, the Secretary of the PRDE determined that all 
science and mathematics curricula be aligned with the SSI; and in Texas, the State Board 
of Education adopted the Texas Essential Knowledge and Skills (TEKS), including TAAS 
testing and accountability system. New teacher certification and licensure regulations and 
new teacher performance requirements were also added. 

4. Resource changes: The Foundation has provided a total of approximately $265 million 
to date to stimulate and catalyze selected, high-quality, systemic reform efforts designed 
to ensure K-I2 high-quality, standards-based mathematics education for all students 
statewide. In FY 1997, the NSF provided the total of $18,092,886 for this purpose— an 
amount that was leveraged with $391.2 million as follows: $71.2 million state funds; 

$281.7 million federal hands, including Title I, Eisenhower, Goals 2000, and Perkins 
hands; $12.4 million local government funds; and $25.9 million other funds. 

Resource changes and leveraging funds are well illustrated by the following examples. 

The Puerto Rico SSI got a State Department of Education’s steady cost-sharing 
commitment increase from $800,000 in 1992-1993, to $3.5 million in 1997 and pledges of 
up to $3.4 million for year 2000. In 1996, Connecticut received approximately $50 
million of ESEA “Improving Basic Programs” handing. This handing was increased to 
over $59 million in 1997. The Initiative informed that while it is currently impossible to 
ascertain the exact amount of these funds that are used for mathematics and science 
education, districts are now held accountable by criterion-referenced, standards-based 
assessments rather that standardized norm-referenced tests. In other words, since 1996, 
all Title I funds currently expended by districts for mathematics and science education are 
directly aligned with the SSI’s goals and objectives. In like manner, the Louisiana SSI 
pointed out that it has leveraged $37 million in state and federal resources since its 
inception in 1991. In addition, the legislature has provided $235 million in new resources 
to cover costs associated with assessment, professional development, accountability, and 
learning technologies; and since 1995, LEARN grants have provided local school districts 
more than $15 million for standards-based curriculum planning, local school improvement, 
and professional development. The Texas SSI’s strategy for convergence of resources is 
to focus the attention of all educational constituencies on the TEKS. During 1997, the 
emphasis shifted from TEKS development and adoption to TEKS implementation. The 
SSI has placed an equal emphasis on the convergence of human and financial resources. 

In 1997, the emphasis shifted to producing resources and professional development 
experiences for use in the implementation of TEKS. The strategy for influencing the 
convergence of financial resources also focuses on the TEKS. At the heart of the strategy 
is a highly successful incentive grant program which requires grantees to provide matching 
funds. This strategy is used in five major SSI projects: pre-service mathematics, pre- 
service science, the Connected Mathematics Project, Title I, and TEXTEAMS. 

In 1997, the New Jersey SSI leveraged $16 for every NSF SSI dollar. To promote equity 
and equal access to technology resources, the State’s new school funding formula 
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designates $40 per pupil for a total of $50 million in Distance Learning Network Aid. In 
addition, the New Jersey Department of Education has allocated $12 million over two 
years to enable all teachers to access technology-related professional development 
opportunities. An additional $10 million has established new Technology Training 
Centers, located in each of the New Jersey’s 21 counties. The State has committed over 
$287 million to improving technology resources over the next five years. Furthermore, 
the Arkansas SSI has leveraged significant amounts of funds, among which $15.6 million 
are state funds through the Arkansas Department of Education; and $15.2 other federal 
funds, including Title I, Eisenhower, and Goals 2000 funds. 

The amount of leveraged funds has certainly increased in the SSI Program during 1997- 
1998 However, there is still the concern about significant amounts of these funds not 
being fully utilized to reflect a unitary mathematics and science program statewide. Title I 
funds, for instance, are still a major source of employment and salaries, and technology 
resources are used in a very piecemeal fashion, not contributing to the development of a 
technology vision, part of the mathematics and science systemic reform fabric. 

5. Management change: Management changes have occurred in the SSI reporting states 
and have strengthened the various systemic reform organizational infrastructures. For 
example, New Jersey, Arkansas, Massachusetts, and Louisiana added science and 
mathematics specialists to their organizations. These specialists oversee and monitor the 
mathematics and science programmatic activities throughout the year, and coordinate with 
other existing state resources and similar efforts. Correspondingly, South Carolina 
added three new positions at the State Department of Education to support SSI efforts; 
and Vermont reported a new coordinator of information technology as well as the 
configuration of standards leadership teams. Overall, overseeing implementation and 
coordination with other ongoing efforts by these new positions, along with the roles 
exerted by the Principal Investigators and Project Directors have significantly contributed 
to the improvement of the operation of the SSI Program. 

6. Data utilization: All the SSI states have designed a system to gather data, either 
through their own evaluation components or by direct communication and collaboration 
with the state departments of education. For instance, state departments of education in 
New York, New Jersey, and Arkansas provide “report cards” to various constituencies 
and have data available to the SSls Puerto Rico indicates that the program evaluation 
and assessment of the SSI is based on a participatory research model that provides for 
multiple measures of key variables that have included student academic performance, 
participant attitudes, participant empowerment, and professional development impact to 
assess the effectiveness of reform. Probably, Vermont is one of the best examples to 
illustrate the importance of data gathering and use. The State was a data-poor site when 
the SSI Program began. In fact, there was a statewide bias against data and a grassroots 
sense that data collation would be an invasion of local control. The scene has been 
radically changed. Today, the VT SSI bases all its improvement work on analysis of data 
and demonstrated need. In schools, assessment truly drives curriculum and districts insist 
in the gathering and analysis of data. The Initiative’s external evaluation component, and 
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the Vermont Department of Education are the major sources for data gathering. 

Likewise, South Carolina is in the process of adding a new centralized, windows-based 
data collection system throughout the State. Louisiana gathers data and insights from a 
variety of sources to increase its effectiveness. These include utilizing national 
mathematics and science education consultants, visiting professional development and 
leadership sites around the State, analyzing national and international reports and journals, 
holding statewide retreats and meetings, and conducting internal evaluations. 

Analogously, the Connecticut SSI informed that the transformation of student 
performance and other educational data into meaningful information for school 
administrators and teachers has become the capstone of the Initiative s systemic 
infrastructure. It has been realized that one of the most significant impediments to 
achieving high-standards teaching and learning confronting school districts is the ability of 
educational systems to move from an almost indecipherable amount of data to a relevant 
and useful information system. Correspondingly, Massachusetts reported that the data 
collection, analysis and feedback process begins with information gathered from school 
districts by Regional Providers and SSI staff in conjunction with other Department of 
Education staff and the outside evaluators. Analysis is performed by the SSI staff working 
with staff from the Department of Education Assessment and Evaluation cluster, with 
assistance from Systemic Research, Inc. 

Data collection does not seem to represent a problem for the SSI states. Nevertheless, the 
effective use of data, including its utilization during the decision-making process appears 
problematic through the SSIs. Furthermore, the disaggreation of data, while required for 
NSF’s reporting does not appear to be instrumental. 

7. Learning infrastructure change: 

(a) Standards-based curriculum: The SSI states reported substantial progress in the 
implementation of science and mathematics standards-based curriculum in a significant 
number of K-12 classrooms. Although curriculum activities vary among the reporting 
states — from focusing in the understanding and implementation of new science and 
mathematics state frameworks (e.g. Connecticut and Louisiana) to the extensive 
implementation of standards-based curricula at the classroom level (e.g., Puerto Rico, 
Arkansas, New Jersey, and Massachusetts) — most of them reflected understanding of 
high-quality curricula, knowledge of the availability of effective curricular materials, and 
the process that is required to implement these materials at the classroom level (e.g., use 
of instruments with appropriate criteria to select instructional programs consistent with 
their science and mathematics education vision). Instructional programs, such as 
Investigations in Numbers, Data, and Space; Connected Mathematics Project; FOSS, 
Science and technology for Chidden; Science Education for Public Understanding 
Program; Chemistry in the Community; and Insights in Biology were mentioned by most 
of the states to represent the generation of new curriculum projects that are being infused 
into the classrooms as a result of the NSF-funded systemic initiative. The Arkansas SSI 
has promoted the use of standards-based curricula in over 73% of the 3 1 1 school districts 
in the State. Likewise, New Jersey has implemented effective curricula in over 55% of 



BEST COPY AVAILABLE 



10 



the school districts; South Carolina has implemented elementary science program in over 
50% of the elementary schools; and 79% of the districts in Massachusetts are piloting at 
least one of the curriculum programs — 64% of the districts are implementing at least one 
program. In Puerto Rico, teams of experienced university faculty from the Natural 
Sciences and Education programs, as well as exemplar classroom teachers, worked 
collegially to review and adopt or adapt national exemplary curricula to meet local needs. 



Information provided during the PER process revealed that in some states there is a 
significant difference between the implementation of standards-based instructional 
, programs in science and in mathematics. Vermont, for instance, evidenced such a 
difference and explained it in terms of a lack of available curriculum materials for that 
subject. Although this difference had been previously acknowledged by NSF, curriculum 
implementation progress had also been identified in both mathematics and science. 
However, the lack of curriculum options in science had never been mentioned as the cause 
of the aforementioned difference in the implementation of curriculum products. 

In spite of the fact that curriculum implementation is a rather successful activity, the 
reporting states evidenced a lack of a coordinated plan to scale-up implementation efforts. 
Some of these states, such as Connecticut and Louisiana have designed effective scale- 
up plans that need to be aggressively implemented within the next year, should they wish 
to expand statewide. The definition of the “unit of change” within the reform theory 
described by the states seems to be an impediment for this purpose. The identification of 
this unit varied from the school district to the teacher, defining a wide range of scale-up 
efforts and action plans. 

(b) Hands-on, inquiry based instruction : Participating SSI states in the PER process 
evidenced hands-on, inquiry-based instruction being used in K-12 classrooms. Overall, 
states documented ongoing professional development programs to provide science and 
mathematics teachers experiences in this teaching approach, particularly through external 
evaluation efforts, including surveys and classroom observations. The Connecticut SSI 
reported technical assistance teams working in the districts, schools, and classrooms to 
ensure that instructional practices are consistent with the districts’ approved curriculum. 
The Puerto Rico SSI informed that the model curricula developed by the Initiative follow 
a constructivist approach to the teaching/leaming process that emphasizes inquiry and 
cooperative learning methods — professional development provided offers first hands-on, 
inquiry-based teaching/learning methodology. The New York SSI promoted inquiry- 
based instruction through the statewide summer institutes and the research and 
demonstration institutes as well. The South Carolina SSI indicated that 90% of the 
surveyed principals responded that the hubs’ support has significantly increased the 
amount of inquiry-oriented, hands-on science and mathematics. Likewise, the Texas SSI 
developed guidelines for the preparation of prospective elementary teachers in both 
mathematics and science that stressed hands-on, inquiry-based instruction. Nevertheless, 
the frequency and amount of the observations made at the classroom level was not enough 
in any of the instances to ensure that the inquiry-based instruction that is promoted 
through the various professional development programs is actually implemented in 
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classrooms. More and better quality data about this indicator need to be gathered and 
interpreted to document this indicator. 

(c) Assessments: Some states reported recently made efforts to develop and adopt new 
assessment systems aligned with the new state standards and frameworks. Efforts of this 

nature are illustrated by the states of New Jersey, South Carolina, Arkansas, and 

Vermont with entire new assessment systems oriented towards the science and 
mathematics standard. Similarly, Louisiana is radically changing its state assessment 
policies and practices, including s. aligning mathematics and science CRTs with the new 
standards-based curricular frameworks, formally adopted in 1996 and 1997. In 1998, 
Massachusetts will adopt the Massachusetts Comprehensive Assessment System, which is 
based directly upon learning standards contained in the curriculum frameworks. 

Moreover, in Connecticut, the Mastery Tests are designed to assess essential standards- 
based subjects, including mathematics. In grade 10, a science assessment addresses major 
goals, including conceptual understanding, applications, and experimentation. Other sites, 
such as Puerto Rico, provide professional development for teachers on standards-based 
assessment to ensure its use at the classroom level. Regardless of these major efforts, the 
need to effectively align state, district, school, and classroom assessments is evident. SSI 
states currently face the challenging issue of using data from non-standards-based 
assessment systems at the state level to evaluate and judge their standards-based efforts at 
the district, school, and classroom levels. 

(d) Student support: Every state documented activities to support students in science and 
mathematics. Texas, for example, implements the Advancement Via Individual 
Determination program designed to give students who ordinarily would not be in rigorous 
academic college-preparatory classes the opportunity to take such classes and the support 
necessary to succeed in them. Arkansas has developed a consolidated state plan to bring 
together a variety of funding sources to benefit student learning which requires the 
coordination of resources from Title I, Eisenhower, Drug Education, and other resources. 
New Jersey’s SSI school districts, such as South Brunswick, have implemented strategies 
to provide special support to students in mathematics and science. This school district 
identified algebra as a critical course for promoting equity, but recognized that at-risk 
students would have initial difficulties with the subject and institutes a summer programs 
to introduce these students to the topic and prepare them for potential pitfalls. 
Nevertheless, the frequency and quality of these programs, especially with regard to their 
direct connection with ongoing systemic reform efforts represent a concern to the SSI 
states. 

(e) Use of environments and resources outside of schools: Some significant level of 
productive activity was reported with regard to the use of informal resources directly tied 
to curricular objectives. In Vermont, it has been essential to reach out to a maximum 
informal science providers, which now serve as a support system to schools and teachers, 
including the Montshire Museum, the Vermont Institute for Natural Science and the 
Vermont Math Coalition, for teacher internships at science and mathematics-based 
businesses statewide. In Connecticut, at least 54 districts have been identified that are 
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providing some type of organized evening, Saturday, or summer educational experience 
that is standards-based. In addition, in Massachusetts, every SSI district is encouraged to 
develop local partnerships with business, local School-to-Work partnership groups, 
museums, higher education institutions, parent groups, and adult education centers. 
Moreover, in New Jersey, systemic improvement districts have seen explosive growth in 
the use of Internet to link students and teachers to outside resources. 

(f) Student-teacher-curriculum interactions: Every reporting SSI state described the 
extent to which changes identified in previous indicators have occurred at the classroom 
level. Vermont, for instance, indicates that evidence has been gathered through the 
school partnership benchmarking process and other data surveys that indicates substantial 
levels of change at the classroom level. Texas’ model of statewide systemic reform 
considers individual schools as the unit of change. School-wide change, however, 
involves more than just the interaction of students, teachers, and curricula. Indeed, they 
believe that the environmental context of the school, as well as the community in which 
the school is embedded, act as critical components in successful school-wide change 
efforts. Louisiana indicated that it is through the interaction, integration, and 
implementation of combinations of changes that reform is realized and sustained. During 
its six year life, the program has built a critical mass of agents for change through the 
teachers and schools involved in the professional development projects and related 
curriculum, technology, and partnership projects. Arkansas reported that observations 
made by staff and the external evaluator, as well as by participants at various Leadership 
Academies, provide evidence that systemic educational changes have been brought 
together at the level of the classroom to benefit student achievement. Massachusetts 
pointed out that changes that have been supported by the SSI in that State have led to the 
writing of the frameworks, played a key role in creating a new assessment system, and 
provided new models and standards for professional development and teacher 
leadership — all of which has accelerated change at the classroom level. Finally, New 
Jersey informed that direct observation reports on classroom interactions in all systemic 
improvement districts reflected significant changes and meaningful improvements. 

(g) System environment/context: All of the SSI states provided information and 
explained how and why the SSI Program is the unitary mathematics and science education 
leading initiative. However, evidence shared on other indicators, such as use of resources, 
for example, did not favor some of the arguments presented. Nevertheless, it can be 
stated that if not the unitary science and mathematics system at the present moment, 
because there are still fundamental issues to address regarding this assessment, all the SSI 
states seem to be moving in that direction. The best illustrations were made by 
Massachusetts and New Jersey, both with an internal/extemal operation system. 

8. Student performance: Almost all of the SSI states reported student achievement gains 
during 1997, mainly by using data gathered from current statewide systems. In 
Louisiana, for example, the Initiative has monitored the Louisiana Educational 
Assessment Program (LEAP) mathematics test data for 5* and 7 th grade students 
identified as having SSI teacher-participants. This system, currently undergoing 
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significant revision, has revealed over five years (1993-1997) that the 5 and 7 grade 
scores of those students identified as SSI teachers’ students outscored students of non-SSI 
teachers. In general, but not in all cases, the score differentials proved to be higher for 
African-Americans and for those on free or reduced lunch status. That the SSI Program 
has particularly benefited traditional “low achievers” is also shown by consistent and 
substantial lower failure rates for SSI students on the LEAP test, especially at the 7 
grade. Connecticut’s student academic performance on multiple state and national 
assessments has continued to show steady gains since 1991. Scores have been increasing 
on a range of indicators, including the CMT, the CAPT, and the NAEP. Since 1993, the 
first year of the improved CMT, 4 th grade student results have improved 6%, from 53% of 
the State’s students meeting goal, to 59% in 1996. In grade six, there has been a 7% 
performance increase, from 45% reaching goal in 1993, to 52% in 1996. Grade eight 
growth, while less, has increased 5%, from 46% of students meeting goal in 1993, to 51% 
in 1996. The CMT student performance increases are occurring within all of the SSI 
districts. The four core districts all show increases in grade four and eight. The increases 
range from 66% in grade four in New Haven, to a 9% increase in Waterbury. The 
Massachusetts SSI informed that MEAP test results in 1996 revealed that between 1992 
and 1996, the percentage of students performing at proficiency level 2 or above increased 
in both mathematics and science at all three grade levels tested (4, 8, and 10). Increases in 
the percentage of students at level 2 or above ranged from 3% in tenth grade mathematics 
and science, to 8% in 4 th grade mathematics and 13% in 4* grade science. At fourth and 
eighth grades, the aggregate improvement for cohorts of districts that had participated in 
the SSI for more than two years was greater than that for the State as a whole, while at 
10 th grade, the SSI districts kept pace with the State in mathematics and slightly outpaced 
the State in science. Likewise, the Puerto Rico SSI indicates that in 1994, the Puerto 
Rico Assessment of Educational Progress, a translated version of the NAEP, was 
administered to a representative sample of 12,000 fourth and eighth grade students 
participating in the SSI Program, regular public, and private schools. The results 
demonstrated that eighth grade SSI students performed significantly better than those 
from non-SSI students and that they reduced in half the performance gap that exists 
between public and private schools in mathematics and by one third the gap in science. 

The 1997 administration results of this system are currently being analyzed. Similarly, the 
Texas SSI informed that students have made exceptional gains in mathematics as 
measured by the Texas Assessment of Academic Skills (TAAS). The percentage of 
students meeting minimum expectations on TAAS mathematics tests for grades 3 through 
8 has increased by at least 30% at each grade level; the percentage has also increased by 
26% at grade 10. 

In Arkansas, as a result of the increased participation of middle-school teachers in the 
Mathematics and Science Crusades, scores improved from 1993 to 1995. Using the 
Stanford 8 achievement test in the spring of 1993 and in the fall of 1995, seventh grade 
students showed improvement from 50 to 53 percentile points in mathematics and from 57 
to 63 percentile points in science. In addition, using the Stanford 9 achievement test data 
from the fall of 1996 and the fall 1997, students in grades 5, 7, and 10 improved their 
mathematics and science performance. In addition, in a study made by the New Jersey 
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SSI’s internal evaluation, several areas were examined with regard to the student impact 
of participation in both K-8 professional development and district systemic improvement 
initiatives. The study reflected that schools involved in neither initiative showed 3.7% 
reduction in percentage of students scoring not proficient on the mathematics component 
of the Early Warning Test (EWT) between 1994 and 1996. Schools involved in 
professional development showed a . 4. 4% reduction in percentage of students scoring not 
proficient on the mathematics; and schools involved in both initiatives showed a 8.5% 
drop in percentage of students scoring not proficient on the mathematics component of the 
same test. South Carolina utilizes the Metropolitan Achievement Tests (MAT7) in 
grades 4, 5, 7, 9, and 1 1 . In addition, the Basic Skills Assessment Program (BS AP) 
assesses student performance on statewide objectives in reading, mathematics, writing, 
and science for grades 3,6,8, and 10. Between 1994 and 1997, the percent of students 
meeting standards on the BSAP Science Test has increased on each of the grades tested'. 
3,6, and 8. In mathematics, between 1995 and 1997, the percentage of students scoring 
above the 50 th national percentile on the MAT7 increased in grades 4, 5, and 7. Also, the 
percent of fourth grade African-American students scoring in the upper quarter increased 
eight percent points form 1995 to 1997. 

Although student achievement gains have been demonstrated, it is important to underscore 
the fact that the testing systems used for this purpose are not standards-based assessment 
systems and that attribution to the SSI Program may seem premature in some instances. 

9. Partnerships: Establishing partnerships and collaborations have been a successful and 
productive SSI activity. Parent organizations, community-based organizations, 
professional associations, business, industry, and the higher education sector have 
responded positively in support of the science and mathematics reform agenda promoted 
by the SSI Program. The development of strong local partnerships that support and 
enhance classroom practice is a basic tenet of the SSI in Massachusetts. On the district 
level, every leadership team is urged to include partners from the business, museum, 
higher education, adult education, and parent communities. All partners participate in 
leadership team planning. Museums and higher education partners lead in providing 
professional development to district teachers. Parent partners and teachers co-train other 
parents to run Family Math and Family Science nights. In the same way, the Texas SSI 
and the Charles A. Dana Center seek and win contracts and grants that shape the manner 
in which large amounts of monies are expended on education and education reform in the 
State. Also, through the Building a Presence for Science Project, a collaboration 
between the SSI, the NSTA, and the Exxon Education Foundation, teachers have 
received professional development on the National Science Education Standards. 
Furthermore, the New York SSI has distributed brochures and other information to 
parents to help them understand how the new science and mathematics standards influence 
classroom learning; New Jersey has developed effective partnerships with 14 higher 
education institutions to provide professional development to K-12 teachers; and South 
Carolina has made significant coordination with the business sector through its 13 hubs. 
Connecticut reported that on August 7, 1997; the State Board of Education adopted a 
new policy statement on “School-Family-Community Partnerships”, which includes a 
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recommendation that schools take the lead in developing programs related to six 
standards, including parenting, communicating, volunteering, learning at home, decision 
making, and collaborating with the community. As another example of effective 
partnership, all major universities are active members of the Resource Center for Science 
and Engineering, an alliance of the major higher education institutions in Puerto Rico, 
facilitating and strengthening the SSI Program. 

Partnerships focus on grassroots community organizations in most of the SSIs. These 
partnerships were established early and have been nurtured and expanded. Each of the 
SSIs also has a significant partnership with at least one major business/industry in the 
state. Outreach has been sought through newspapers, television programs, and direct 
dissemination to parents. 

• g r am fees Is Implications attd Emerging Issues 

The following issues emerged from the analysis of the 1998 SSI PER process: 

Theory of reform, the vision, and the unit of change: The majority of the SSI states 
explained the theory of reform that supports the systemic science and mathematics reform 
agenda. While describing the vision of science and mathematics education at various 
levels (e.g., classroom, school, system), each SSI identified its unit of change. 
Massachusetts, New Jersey, Connecticut, and Arkansas defined the school district as 
the fundamental systemic reform unit of change, whereas Puerto Rico, Texas, Vermont, 
New York, and South Carolina identified the school as that unit. Only one state, 
Louisiana, defined the teacher as the unit of change. 

Despite the aforementioned units of change, the descriptions of the science and 
mathematics education visions consistently reflected the same elements. Nonetheless, 
assisting the states to define the ultimate unit of change is an essential step that the SSI 
Program needs to take in the very immediate future. Operationally, the identification of a 
given unit of change (e.g., school, school district, teacher) demands a particular action 
plan and defines a unique scaling-up plan as well. Defining the unit of change as that 
element of the system that must be changed as a necessary condition for altering content, 
teaching, and assessment in classrooms (e.g., the teacher) is considerably different from 
stating that the unit of change is the component of the system that must be changed in 
order to assure self-sustained support and accountability of systemic reform beyond the 
life of the SSI (e.g., the school district). 

Equity: Other than describing the science and mathematics vision of high-quality 
education for all students in a rhetoric fashion in most of the instances, scarce reference, if 
any, was made during the PER process to the SSIs’ particular efforts to ensure equitable 
participation of every student in excellent science and mathematics education. Almost no 
information or evidence was provided in terms of specific strategies or measures taken to 
infuse equity throughout the entire initiatives and reflect this critical element in every 
single component of the systemic reform enterprise. Equity, as a central and essential 
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element of reform needs to be moved from its current “add-on’ status to an inherent 
constituent of the entire science and mathematics educational reform activity. 

The SSI as the science and mathematics unitary program: SSI states evidenced 
progress in the area of leveraging resources and converging them towards the mathematics 
and science unitary program that they are attempting to implement statewide. However, 
stronger policies and measures need to be in place to ensure that this happen, especially 
with regard to the use of categorical funds, such as Title I. In some states, the information 
provided indicated how much of these funds are being used under the SSI Program, but 
also suggested the utilization of significant amounts of these funds for purposes other than 
those promoted by the SSI. 

Transition to standards-based science and mathematics curricula: In spite of the fact 
that most of the SSI states have made significant progress in the implementation of 
standards-based science and mathematics curricula to date, the statewide impact is still 
limited. Even those states that have made notable progress in this respect have only 
reached slightly over 50% of the school districts with full implementation of standards- 
based curricula. States such as Louisiana and Connecticut are beginning to make some 
progress in this direction; Massachusetts and Puerto Rico, with the most notable 
progress in this area, have to scale-up rapidly in order to reach a notable portion of the 
student population. A similar situation is faced by Arkansas, Vermont, New Jersey, and 
South Carolina. In addition, the process of transitioning to standards-based curricula 
requires that schools, as fundamental units of change, conceptualize and develop the 
school science and mathematics program before beginning the process of selection and 
implementation of standards-based curricula. 

Transition from micro-activities to macro-activities (Scaling-up): Every SSI needs to 
address the issue of extending its initial efforts in order to become a statewide enterprise. 
Moving from an initial, predetermined number of schools or school districts to a larger, 
significant number of school or school districts capable of implementing a standards- 
based, high-quality mathematics and science for all students requires a well-designed 
strategic plan and represents a major challenge for all existing SSls. 

Infusion of technology into science and mathematics instruction: SSI written reports 
and oral presentations during the 1998 PER process revealed that states still need to 
develop a vision of technology as an inherent component of systemic reform. The use of 
technology as a tool to advance the understanding of mathematics and science at the 
classroom level was not clearly conveyed or evidenced by the SSI states. Rather, the 
notion of a collection of isolated technology projects and strategies to experiment with 
various aspects of technology seemed to be a recurring pattern. 

Standards-based assessments: Regardless of the progress made by SSI states in terms 
of the use of standards-based assessments to gather student achievement data aligned with 
standards-based curriculum and instruction, this issue is still one that requires special 
attention. Some states have very recently designed or adopted new assessment systems 
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aligned with the national standards that will eventually provide more reliable student 
performance data. Nevertheless, the misalignment among assessment systems used at the 
state level, district-wide, school-wide, and in classrooms prevails. The understanding of 
assessment standards as well as the design, development, adoption, and effective use of 
standards-driven systems and tools need to be aggressively addressed. 

Evaluation (Internal/External): Although substantive evidence and documentation 
related to accomplishments by the SSI states were presented during the 1998 PER 
process, evaluation, overall, is viewed as an ongoing issue. It was not totally clear, from 
the information provided by participating states, how the data gathered throughout the 
year regarding the various components of any of the systemic initiatives are effectively 
utilized to facilitate learning at the classroom level. Moreover, the ultimate use of student 
achievement data, even among those states that have significantly evolved in this direction, 
is unclear. How does this information flow into the classroom and how it is used by 
administrators and teachers is still an issue. Likewise, documentation and relevant data in 
terms of teachers behaviors in classrooms as a result of the massive professional 
development programs being promoted by the SSIs is yet to be improved, both in quantity 
and quality. 
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Guidance to Cohort UI SSIs as a result of the PER findings: The following major 
systemic reform issues are viewed as emphases for the Cohort-Ill SSIs during FY 1998: 

1. Accomplishment of the transition from capacity building or infrastructure to full 
implementation, including an assessment of the systemic reform “enablers”, such as 
policies; the mathematics and science standards-based curriculum frameworks; 
standards-based student assessment systems; aligned and distributive professional 
development processes; accountability measures; and resource convergence to yield a 
unitary mathematics and science education reform process. 

2. Assessment of the following specific, fundamental issues that must be successfully 
addressed by any SSI at the completion of its fifth year of implementation: 

• based on implementation to date, strategies (with explicit measures of 
achievement) to fully implement the mathematics and science standards-based, 
inquiry-focused curricula throughout the classrooms of the participating schools of 
the state’s districts; 

• institutionalization (process, deliverables, and milestones) of the aforementioned 
mathematics and science curricula for all students (equity is embedded in the 
requisite educational processes) via use of appropriate mathematics and science 
instructional materials, educational resources, and the K-12 mathematics and 
science standards-based curriculum frameworks; 
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• the appropriate incorporation of learning technologies into mathematics and 
science instructions; 

• institutionalization (process, deliverables, and milestones) of standards-based 
mathematics and science student assessment systems; 

• systemic-reform-directed professional development strategies to yield the total 
mathematics and science instructional workforce consistent with full 
implementation throughout the system; 

• strategies (with deliverables) to assess both the progress of and productivity of 
systemic reform of the extant system; and 

• the distributive organizational process/structures for the achievement of rapid 
replication, innovation, and overall system- wide or state- wide scale-up. 

Assistance to Phase II SSIs: Guidance and special assistance to the Phase II (Cohorts I 

& II) SSIs will focus on the following areas: 

1 . Assist the states to implement and accomplish the following sets of activities, 

consistent with NSF/ESR’s systemic reform expectations: 

• broadly- based strategies to institutionalize the systemically reformed, 
standards-based, inquiry-centered, math and science education instructional 
program (a unitary activity) throughout the classrooms of the participating K- 
12 schools of the state's districts; 

• strategies to develop a professional continuum to teacher development, 
integrating recruitment to a teacher education program; preservice preparation; 
teachers licensing; new teachers induction; ongoing professional development; 
and advanced certification of teachers based on performance. 

• progressive and appropriate infusion of learning technologies into math and 
science instructions throughout the participating entities of the SSI; 

• full implementation of standards-based mathematics and science student 
assessment(s) independent of local and state student assessment requirements 
unless each is appropriately standards-based; 

• strategies to better assess the orderly progression toward maturation and the 
unit productivity of systemic reform of the extant system, including full 
accommodation of specificity as its bears on the strategic actions- outputs 
continuum; 
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• the distributive organizational processes/structures necessary for the 
achievement of predictable and orderly innovation replication and overall 
system-wide or state-wide scale-up to yield a fully reformed (mature), 
standards- based, high-performance system effectively serving of the 
(participating) students upon the completion of Phase II of the respective SSI; 

• development of the capacity to demonstrate/disseminate best practices, results, 
procedures, products, and services in order to assist other states with the 
replication of these outcomes in a systemic fashion: 

(a) identification, evaluation, selection, and implementation of 
standards-based curriculum; 

(b) identification and selection of standards-based assessment 
systems; 

(c) design, development, and implementation of a standards-based, 
research-centered professional development program; 

(d) alignment of curriculum, professional development, and 
assessment; 

(e) scaling-up strategies; and 

(f) establishment of policies and measures to support systemic 
reform; and 

• ensuring high-quality of the entire enterprise. 

The Vermont SSI: As a result of the 1998 PER, ESR projects to take various actions in 
order to ensure that SSI states successfully achieve the objectives of their science and 
mathematics reform agenda. In addition to the special assistance that will be provided to 
both Phase II states and Cohort III sites, ESR will conduct a comprehensive evaluation of 
the SSI in Vermont to assess the Initiative’s infrastructure and to ensure its capacity to 
transition to full statewide implementation. The assistance and evaluation of the VT SSI 
will include the following aspects: 

(i) the design and utilization of broad-based strategies to fully implement the mathematics 
and science standards-based, inquiry-focused curricula throughout the classrooms of 
the participating school districts; 

(ii) institutionalization of the aforementioned mathematics and science curricula for all 
students; 

(iii) institutionalization of standards-based student assessments; 

(iv) strategies to assess both the progress and productivity of systemic reform of the 
extant system; and 

(v) capacity to replicate successful strategies, products, and processes statewide. 

The need for an SSI Impact Study: ESR plans to initiate the design and implementation 
of a study aimed at describing and explaining the connection among the SSI infrastructure, 
the transition from that infrastructure to full implementation and scale-up, and student 
achievement. The following preliminary outline is being analyzed: 
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A Proposed Statewide Systemic Initiative (SSI) Program Evaluative S tudy 

1. Assessment of the progress achieved to date with regard to the original objectives of 
the SSI Program as stated in the FY 1990 Program Solicitation. 

A. To encourage improvements in science, mathematics, and technology 
education through comprehensive systemic changes in the education 
systems of the states. 

B. Encourage coherent and consistent policies and programs by identifying 
the elements that, if taken together, will make a difference in what 
students know and are able to do. 

1 . A vision of science and mathematics education 

2. curriculum goals and content 

3. instructional strategies 

4. assessment of student learning 

5. changes in school structure and decision-making to promote 
effective teaching and learning 

6. equity as a central and inherent issue to systemic reform 

7. policy changes in support of high-quality science and 
mathematics education 

8. adoption of new mew methods and standards for the 
preparation and continuing development of teachers and 
administrators 

C. Creation of effective partnerships 

D. Plan for implementing and evaluating results 

E. Determining impact on student achievement 

II What have we accomplished to date in terms of each of the systemic reform 
components include in the “drivers”? 

A. curriculum 

B. assessment 

C. professional development 

D. policies 

E. partnerships 

F. resources 

G. data gathering, interpretation, and use 

H. student achievement 

I. equity 

J. scaling-up 

K. institutionalization 

III. Validity and Attribution of Student Achievement Data 

A. Progressive increases in student mathematics achievement 

B. Progressive increases in student science achievement 

C. Statistical analyses of student achievement data 

D. Selective impacts of the SSI Program 
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E. Validity of the SSI attribution 



IV. Lessons Learned 

A. What have we learned about systemic, reform? 

B. What works and under what circumstances, especially with regard to • 
the following systemic reform components and the relationship and 
dependence among them? 

1 . curriculum 

2. assessment 

3. professional development 

4. policies 

5. partnerships 

6. resources 

7. data gathering, interpretation, and use 

8. student achievement 

9. equity 

10. scaling-up 

1 1 . institutionalization 

V. How can we share our learnings with other states to assist them reforming their 
educational systems through a systemic approach? 

A. What are the fundamental elements? 

B. What is the most effective process? 

C. How would the “how-to-do-it” kit look like? 

The need for a plan to ensure sustained interaction with post-SSI awardees: The 

SSI Program was initiated in FY 1991 with 10 states. A total of 25 states and the 
Commonwealth of Puerto Rico received SSI awards for terms of up to 5 years of funding 
and of up to approximately $2.0 million per year. As the states completed their 5-year 
agreement with NSF, the Phase II competitions (to date, NSF has conducted the Phase II 
process with Cohort- 1 and Cohort-II SSls; proposals for Cohort-Ill states are due on 
February 2, 1998), and the phase-out process, the SSI FY 1998 includes only 11 SSI 
states. 

The Foundation plans to continue relationships with SSI awardees as they complete their 
cooperative agreements and afterwards, as follows: 

Continuing Contact : The NSF has made a significant investment in each of the SSls. 
Each has made significant progress in the areas described earlier. While this progress 
is not always complete, it has resulted from considerable efforts, and should prove 
valuable to each state in its continuing efforts to improve its K-12 education. The 
Foundation believes that it will serve well all of the state, city, and regional systems 
with which it is working to maintain contact with SSI states, even when no further 
NSF support for their SSI as a whole may be forthcoming. Therefore, ESR will 
maintain contact with each of the SSI states that desires it. For example, each will be 
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invited to PI meetings; receive communications sent to the SSIs; and be kept as an 
active member of electronic and other networks. 

2. Component Development: The SSIs have developed many useful means for 
improving K-12 science and mathematics education in their states. In many cases, 
components have begun under the SSI that could not be completed given its resources 
or duration. The ESR program directors will work with SSIs desiring it to consider 
other sources of EHR support for completing or extending components of the SSI. 
These sources may provide the means by which major portions of the SSI agendas in a 
state could be completed. 

Existing Programs . When a state has begun one or more activities under the SSI 
award and desires to continue it, the state's NSF SSI program director will, upon 
request, facilitate discussions with program directors in EHR who work in the 
relevant program Such common needs as teacher enhancement, teacher 
preparation, and materials development are already organized as programs within 
EHR. 

Cross-program Support . There is also a mechanism within EHR for considering 
proposals that cut across its existing divisional units. A state interested in this 
possibility should consult with its SSI program directors for advice about this 
process. 

3. Continuing as Partners with Others : ESR will also encourage the involvement of the 
SSIs in the various reform programs underway in their states, including the Urban 
Systemic Initiatives (USI); the Rural Systemic Initiatives (RSI); Local Systemic 
Change Projects; Collaboratives for Excellence in Teacher Preparation; or Networking 
Infrastructure for Education programs, as well as reforms supported through other 
federal agencies; through private foundations, business, and industry; and by the states 
themselves. 

4. Completion of Exemplary Systemic Development: While EHR does not deem it 
desirable that even exemplary efforts depend on NSF support indefinitely, it is 
recognized that any a priori period of time is at best an estimate of how long systemic 
reform may take. As a consequence, we are prepared to continue support to systemic 
initiatives that have made substantial progress until the sustainability of the initiative is 
highly probable. EHR is not prepared to support, however, a steady-state reform 
effort. 

The Foundation will design and implement a plan including other strategies and activities 

that may ensure the accomplishment of this objective in a more effective manner. 



V; Appendices 



A. The 1998 PER indicators : The FY 1998 PER indicators as described in Dr. Luther 

S. Williams, Acting Director, Division of Educational System Reform and Assistant 

Director, Education and Human Resources, memorandum of November 3, 1997 (Re: 

Fiscal Year 1998 Program Effectiveness Reviews) are: 

1. Student impact: number of students reached presently with full implementation 
(based on drivers) and percentage relative to baseline number of students in system; 
number and percentage of students presently affected by global system changes. (Core 
data elements submissions I and II should be used in responding to this indicator; 
emphasis is on full implementation of a standards-based, inquiry-centered math and 
science education instructional program and that the percentage of students 
participating in such a fully implemented system relative to the total student pool 
defines the scale of the systemic reform impact.) 

2. Teacher impact : number of teachers involved presently in full implementation 
(based on drivers) and percentage relative to the baseline of the number of teachers in 
the system, number and percentage of teachers affected by global system changes. 
[Teacher impacts should be explicated as a continuum of systemic activities relative to 
the baseline on year one; the core data elements submissions would be useful inputs 
for this indicator. Moreover, the progressive transition from ill-defined, short term to 
research-informed, intensive (60 hours or more) to sustained professional development 
in a systemic domain coupled to student performance is another crucial issue for this 
indicator] 

3. Policy changes: m/s graduation requirements; removal of remedial courses; 
professional development requirements; teacher certification changes; accountability 
measures for all system levels, other policies which impact implementation. (It would 
be useful to frame the policy changes in a chronology in parallel with specific program 
objectives/actions.) 

4. Resource changes (both increase and alignment): increase in percent of budget; 
how are targeted funds such as Title I, Perkins, Eisenhower spent and percentage of 
total of each used in direct support of SSI, USI, or RSI; other measures could include 
things such as change in student teacher ratio; use of other professionals inside and 
outside system; facilities including math and science learning technologies. 

5. Management change: cabinet level position overseeing implementation; increase in 
m/s supervisors, coordination of other parts of the system with SSI, USI, or RSI. 

6. Data utilization: data collected relative to implementation; who collects; evidence 
that data is used in formative evaluation and change in strategic plan; validity of 
attributions based on data interpretations/use, site-specific program evaluation. 
(Reference to the data reported in the core data element submissions is recommended.) 

7. Learning infrastructure change: 

(a) Standards-based curriculum: what is the evidence that the curriculum is 
standards-based (should include description of what it is); extent of its 
implementation in classroom; evidence of ancillary support including provision 
of materials, support from principals, lead teachers, changing in school 
schedule needed for effective delivery. (Framing the response as a fractional 







24 



representation of the total system being impacted would enhance the specificity 
of the response.) 

(b) Hands-on, inquiry based instruction: evidence that teachers are being 
appropriately trained in this pedagogy; evidence that this is occurring in 
classrooms; support for teachers in effective implementation including things 
such as teacher planning time, mentor teacher assistance, opportunity for 
continuing professional development. 

(c) Assessments: what assessments are used; alignment with curriculum and 
instruction; training of teachers in their use; assessment feedback to teachers so 
that they may use to individualize instruction; other state or district mandated 
assessments notwithstanding, explain the math and science standards-based, 
instruction derived, assessment system used for the SI. (Stated differently, 
what standards-based math and science assessments are used? If none, why 
not and what are the near term plans to appropriately assess the student 
performance outputs of the systemic reform enterprise?) 

(d) Student support: in-school and other programs to support students as 
curriculum and course requirement changes are being implemented. 

(e) Use of environments and resources outside of schools: extent of utilization 
of informal science resources directly tied to curricular objectives; internships 
and similar programs; outside human resources, other community-based 
resources. 

(f) Student-teacher-curriculum interactions: indicators a-e speak to changes in 
components of the learning infrastructure. For this indicator, explicate the 
evidence detailing the extent to which these changes have been brought 
together at the level of the classroom as described in your description of the 
classroom implementation. 

(g) System environment/context: place the SSI, USI, or RSI in the context of 
your entire system. To what extent is the initiative the unitary mathematics and 
science education system? How does it relate to vocational education, special 
education, Title I, and other targeted programs? Detail the number of students 
in these programs and the congruence of their curriculum, instruction and 
student assessment, graduation requirements and quality of the mathematics 
and science experience to the SSI, USI, or RSI. 

8. Student performance: description of vehicles that are used to measure; evidence that 
they are valid; delta of student performance relative to a baseline at the start of the 
award with a discussion of the delta for the underserved relative to all students; 
alignment with changed curriculum and instruction, career choice information; job 
upon graduation or college attendance; graduation rates. (Employ the core data 
element II documentation and any other assessment/student achievement and 
accomplishment data in responding to this indicator.) 

9. Partnerships: evidence of collaboration with parent organizations, community-based 
organizations and business-industry, and higher education in support of the reform. 

10. Summary Presentation: Based on the responses given to the categorical indicators 
1-9 above, provide summary representation of the progress (to date) toward the 
achievement of a unitary, standards-based, inquiry-centered, math and science 
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education system and justify this representation by use appropriate 
output/outcome/impact measures of systemic reform in a SSI, USI, or RSI site. [This 
response should include qualitative and quantitative (systemic metric) as appropriate.] 
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B. List of awards : The following is a list of the SSI awards, including award 
number, title. Principal Investigator, Project Director, and total funding: 

• Arkansas Department of Higher Education, ESR-93 50027 

PI: Edward Crowe 

PD: Suzanne Mitchell 

TF: $9,750,000 

PO: Julio E. Lopez-Ferrao 

• Connecticut Academy for Mathematics, Science & Technology, Inc., ESR-9634086 

PI: Steve Leinwand 

PD: Richard Cole 

TF: $2,700,000 

PO: Celestine H. Pea 

• Louisiana Statewide Systemic Intiative (LaSiP), ESR-9634088 

PI/PD: Kerry Davidson 

TF: $2,800,000 

PD: L. Jody Chase 

• Massachusetts Department of Education, ESR-97 1 2003 

PI: David Driscoll 

PD: Thomas Noonan 

TF: $2,600,000/24 month award 

PO: Celestine H. Pea 

• New Jersey Statewide Systemic Initiative, ESR-9350023 

PI: Gerald Goldin 

PD: Deborah Cook 

TF: $9,999,991 

PO: Julio E. Lopez-Ferrao 

• New York State Board of Education, ESR-9350033 

PI: Edward Lalor 

PD: Richard Jones 

TF: $10,000,000 

PO: Gerald E. Gipp 

• Puerto Rico Statewide Systemic Initiative, ESR-97 1 1999 

PI: Manuel Gomez 

PD: Pablo Rivera 

TF: $2,560,000 

PO: Gerald E. Gipp 





27 



South Carolina Statewide Systemic Initiative, ESR-93 50042 
PI/PD: Marc Drews 
TF: $10,000,000 

PO: Julio E. Lopez-Ferrao 

Texas Systemic Initiative, ESR-97 12001 
PI: Uri Philip Treisman 

PD: David Hill 

TF: $9,600,000 

PO: Linda S. Warner 

Vermont Statewide Systemic Initiative, ESR-97 1 1997 
PI/PD. Frank Watson 
TF: $1,072,712 

PO: Julio E. Lopez-Ferrao 
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